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FOR CONTROLLING GRAPHICS A? 
vroEO DATA IN MULTLMEDU DA 
PROCESSING AND DISPLAY SYS^MS 

TECHNICAL FIELD OF THE IN^NTION 

The present invention relates in gwferaJ to multimedia 
processing and display systems dSidW particular to appara- 
tus, systems and methods for controHing graphics and video lo 
data overlay in muliimedia processing and display systems. 

CROSS-REFEREN/^ TO RELATED 
APPLICATIONS 

The following copendir^ and coassigned United Stales 
patent applications comwn related information and are 
incorporated herein by^ferencc: 

U.S. patent applicatton Ser. No. 08/098,846 (Attorney's 
Docket No. P35 10-^1 US), emided "System And Method 20 
For The Mixing O^raphics And Video Signals,'* and filed 
Jul. 29. 1993; 

U.S. patent a^licaiion Ser. No. 08/223,845 (Attomcy*s 
Docket No. P^10-P21US), cniided "Apparatus, Systems 
And Mctho^For Processing Video Data In Conjuncdon 25 
With A Mul^Format Frame Buffer," and filed Apr. 6, 1994. 

^CKGROUND OF THE INVENTION 

As muldmedia information processing systems increase 3Q 
in popularity system designers must consider new tech- 
niques for coMToUing die processing and display of data 
simulianeously\aeneTatcd by multiple sources. In particular, 
there has been ^bsiantial demand for processing systems 
which have die oapability of concurrendy displaying both 
video and graphiS data on a single display screen. The 
development of such systems presents a number of design 
challenges, not only Because the formal dififerenccs between 
graphics and video ma must be accounted for, but also 
because of end user drmn requirements that these systems ^ 
allow for flexible maniMaiipn of die data on die display 




simultaneously displaying 
display screen involves 
case, a stream of data 43 
a display within 
the display soeea to the 
streams defining a dis- 
tg to the same region of 



One particular tec 
video and graphics 
the generation of 
from a selected source 
a particular region or 
exclusion of any non-sel jbted 
play or part of a display conespoL „ 
die screen. The selected data stream\ciicratiDg the display 50 

window "overlays" or "occludes** the^data from the nonse- 
iected data streams which lie "behind" me displayed data. In 
one instance, die overall content andYppeaiance of die 
display screen is defined by graphics daik and one or more 
**vidco windows" generated by data frofta video source 
occlude a corresponding region of thai mphics data. In 
other instances, a video display ot window xAw be occluded 
or overlaid by graphics data or even another vWco window. 

In the muldmedia environment, the *^dowing" 
described above yields substantial advantages. Atongodicr 60 
diings, the user can typically change the size and Ideation on 
die display screen of a given window to flexibly miaipulate 
the content and appearance of the data being display For 
example, in the case of combined graphics and vicUK), die 
user can advantageously create custom composite yi^^ 
displays by combining multiple video and graphicsYaia 
streams in windowing environment 
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In ordflt to eflScicntly control windows in a multimedia 
environme\t eGBcient frame buffer management is required. 
SpecificallyXa frame buffer control scheme must be devel- 
oped which ^ows for the efficient storage and retrieval of 
multiple typesV^daia, such as video data and graphics data, 
to be cost compedtive as well as functionally efficient, such 
a scheme should Vnimize the number of memory devices 
and the amount o^ontrol circuitry required and should 
insure thai data Qow\p the display is subjected to minimal 
delay notwithstanding\data type. 
One of the major (Mculties in managing video in a 



combined video and 
results from the fact that 
displayed are constantly b 
thirty frames per scconi * 
nonnally generated once 
then remain static until ' 
display. Thus, the ocdusio 
graphics data requires that 
of the video data not be di , 
window is updated A second 
systems operating on both video 



windowing environment 
data being received and 
1, typically at a rate of 
the graphics data are 
iphics display and 
that graphics 
of video data with 
'graphics data "in front 
^ed each time the video 
[DCcm with windowing 
graphics data is the 



formatting differences between the ^dco and graphics data 
themselves since video is typically mprized into a YUV 
color space while graphics is digitized into an RGB color 
space. Hence, any combination i^ideo anfci graphics window- 
ing system must have the capability of e^ciemly handling 
data within both the YUV and RGB fo 

Thus, due to the advantages of windowing, the need has 
arisen for efficient and cost effective win&wing control 
ciicuitiy. Such windowing circuitry sbouldVaUow for the 
simultaneous processing of data received nom multiple 
sources and in multiple formats. In particular, suth window- 
ing control circuitiy should be capable of cfficdenily and 
inexpensively controlling the occlusion and/or \veiiay of 
video and gi^faics data in a windowing environ 
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The principles ol 
for the flexible cc 
display control en^ 
video data, grs^cs 
stored in on-screen mei 
generadon of the conespoi 
graphics or video data 
mensory and retrieved wl 
display reaches the 
video window. The window^ 
can then be overlayed over 
on-screen memory under one 
mode, pixels firom the off- 
when the raster scan has 
selected for the window. In a 
is rastered firom the off-screen 



present invention in general provide 
of graphics and video dm in a 
In particular, an entire &ame of 
a combination of both, may be 
and rastered out with the 
ig display screen. A window of 
be stored in off-screen 
ter scan generating the 
on the display for the 
off-screen memory 
rastered out of the 
itditions. In a first 
are rastered only 
isition on the display 
window of data 
hen the display 



raster scan has reached the display wiruftpn/ position and 
graphics data being rastered from the on-9creen memory 
matches a color key. In a third mode, the wMow data is 
rastered out of the off-screen memory when th^aia being 
ou^xit from the on-screen memory matches the\3lor key, 
notwithstanding the position of the raster scan. 

According to a first embodiment of the present inVlpition, 
a graphics and video controller is provided which i 
a dual aperture interface, each word associated wid 
address to a selected one of on-screen and off-screen r 
of an associated unified firame buffer as either graphics < 



video pwccl daia. CircuiLry is provided for writing a word of 
ihc pixS data received by ihc interface lo a one of the 
on-5cree\and off-screen memory areas corresponding lo the 
address as\ociated with the received word Circuitry is also 
included fok selectively retrieving graphics and video data 
from the or^creen and off-screen memory areas. A first 
pipeline is prVided for processing graphics data retrieved 
from the framcvjuffcr and a second pipeline is provided for 
video processing data retrieved from the frame buffer. 

According to aVecond embodiment of the present inven- 
tion, a controller iaprovided which includes a dual aperture 
port for receiving vkleo and graphics data, each word of the 
data received with an address associated directing the word 
to be processed as ^ther graphics or video data and off- 
screen memory spacqs of a frame buffer. A second port is 
included for reccivini real-time video data. Circuitry is 
provided for generailpg an address associated with a 
selected one of the mcnipry spaces for each word of received 
rcal-dmc video data. Cincuitry is included for writing sclec- 



10 



lively the words into 
spaces of the frame 
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-screen and off-screen memory 
^Circuitry is also provided for 
I of data from the on-screen 
i rasiered for driving a display, 
\ ones of the graphics 
buffer. A video 
ssing ones of the 
frame buffer, the 
ng a stream of graphics 
graphics pipeline and 
I backend pipeline when a 
position of a video 
eluded for selecting for 
from the graphics 
I output from the video 

35 



selecuvely retrieving 
and off-screen space^ as 
A graphics backend 
words of dau retrii 
backend pipeline is 
video words of data 
circuitry for retrieving! 
data from the frame 
rastering video data to 
display raster scan 
window. An output 
output between words of data 
backend pipeline and words of 
backend pipeline. 
According to 

a third embodiment of Uie present invention, a 
display system is provided which incudes first and second 
parallel backend pipelines. A multiVfonnat frame buffer 
memory is included having on-screen W off-screen memo- 
ries each operable to simultaneously smre data in graphics 40 
and video formats. A dual aperture port is provided for 
receiving both graphics and video dataias directed by an 
address associated with each word of oata received Cir- 
cuitry for writing is included for wridng a word of video or 
graphics data into a selected one of tne on-screen and 
off-screen areas of the multi-frame buffet Wemory control 
circuitry controls the transfer of data betwVen the first and 
second [back-end] backend 

pipelines and the frame buAcr. The system 
further includes a display unit and overlay control circuitiy 
for selecting for output to the display unitVbetween data so 
provided by the first backend pipeline and dat| provided by 
the second backend pipeline. 

A fourth embodiment of the presem inventii 
a display data processing system which inclui 
writing data into an on-sciten space of a 
circuitry for writing data imo an off-screen 
frame buffer. A video pipeline is provided f 
video data output from a selected one of the on-i 
off-screen spaces. The video pipeline includes a 
in/fint-out memory for receiving selected pixel 
the selected space. The video pipeline also includes 
first-in/first-out memory disposed in parallel to 
first-in/first-oui memory for receiving other sei 
from the selected space in the frame buffer. An inti 
is provided as pan of the video pipeline for gi 
additional data by interpolating data output from the firsi 
second first-in/first-out memories. A graphics pipeline is 
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disposed in parallel to the video pipeline for processing 
gralhics data output from a selected one of the on-screen 
andbff-screen spaces. Finally, an output selector is provided 
for saecting between data output from the video pipeline 
and dab output from the graphics pipeline. 

The principles of the present invention allow for the 
constructron of circuits and systems with substantial advan- 
tages over\he prior art. Among other things, the principles 
of the present invention allow both graphics and video data 
to be stored ma single unified frame buffer and retrieved 
therefrom in aYumbcr of different ways. For example, a 
combinaiion of Jpphics and video data may be stored in the 
on-screen memoW and simply rastered out during screen 
refresh. In anothe^as^r^ entire screen of graphics or video 
data may be stored Wthean-screcn memory while a window 
of graphics or videi data/is stored in the off-screen portion 
of memory. The viAdow data can then be rastered out to 
selectively ovcriaj/^ J^^feoil^ the data being rastered out of 
The m^eriay may be controlled by 
positi|bn with a match of the on- 
it and a color key, or both. 
orAtSprescnt invention provide for the 
ve overlay of video and graphics data 
It In particular, the use of color 
overlay of data in a window 



the on-screen me 
either window d^Ala: 
screen data being 

The cm 
efficient and inex[ 
in a windowing environmi 
comparison to determine 



region elimi n ates the need for precise x- and y-posidon Hat^ 
for the location of that winobw and allows for video crop- 
ping to be performed. Further, the use of graphics data to 
control oveday provides suDstantial advantages in thai 
graphics data is less subject Vo the graininess and noise 
problems often found ^with vidVo data. Further, the user is 
given total conm)t of overlay operations when keying on 
graphics data b ec a u se the gr^hics data is computer gener- 
ated, whmas the video data is c^tured rfaf^ 
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For a more complete imderstandifag of the present inven- 
tion, and the advantages thereof, re&eoce is now made to 
the following descripdons taken in)pnjunction with the 
accompanying drawings, in which: 

FIG. 1 is a top levd functional bkck diagram of a 
multi-media processing and display systeun embodying the 
prindpLes of the present invention; 

FIG. 2 is a more detailed functional blo&k diagram of the 
VGA controller depicted in FIG. 1; 

FIG. 3 is an expanded functional block biagram of por- 
tions of the controller of FIG. 2 with emphasis on the overiay 
control features; 

RG. 4A is a detailed functional block diaftam of a first 
embodiment of the color comparison drcuitiywf FIG. 3; 

RG. 4B is a detailed functional block diagramW a second 
embodiment of the color comparison circuitry of mG. 3; and 

RG. 5 is a detailed functional block diagram ofa selected 
one of the video window position control circuits depicted in 
RG. 3, 
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DESOUPnON OF THE INVENTION 

RG. 1 is a high level functional block diagramXof a 
multi-media processing and display system 100 operame to 
process and simultaneously display on a single dismay 
screen both graphics and video data according to the psj 
ciples of the present invention. Display system 100 indu 
a central processing unit (CPU) 101 which controls 
overall operation of system 100 dod generates graphics < 




defining graphics images lo be displayed. CPU 101 com- 
tunicaics with ihc remainder of the system discussed oelow 
L local bus 103. System 100 also includes a rcal-timc 
CO data source 104. A real time video stream may be 
nrcWed to the system VGA controller 105 in one of two 5 
wayX First, video data source 104 may be coupled to local 
bus ife and a video data stream introduced through dual 
apcrtuft addresses. In this case, video source 104 will 
dirccUyVddrcss the system frame bufifer 107. Second, video 
source 1U4 may be coupled directly to VGA controller 105 
via a deAcatcd bus 109 or "video port." In this instance, 
VGA con\oller 105 generates the required addresses mto 
frame bufffc 107. Real-lime video source 104 may be, for 
example, a Ad ROM unit, a laser disk unit, a videotape unit, 
television caftlc ouUct or other video data source outputling 
video data inV YUV format. CPU 101 operates in conjunc- 
uon with a sy\tem memory 108 which stores graphics and 
video data on ieal-ume basis. System memory 108 may be 
for example raidom access memory (RAM), floppy disk, 
hard disk or otlfcr type of storage device. 

A VGA contiluer 105 embodying the principles of the 
present inventioAis also coupled to local bus 103. VGA 
controller 105 wi\ be discussed in detail below; however, 
VGA controller 105 generally interfaces CPU 101 and video 
llsplay unit 106 and a mulliformat system 
imc buffer memory 107 provides tem- 
! graphics and video data during pro- 
f on display unit 106. According to the 
~ nvenlion, VGA controller is oper- 
siorc graphics and video data 
|l07 in their native formats. In a 
I frame buffer area is partitioned 
' and off-screen memory. Frame 



source 104 with a 
frame buffer 107, 
porary storage of 
cessing prior to dii 
principles of the 
able in selected 
together in fi 
preferred embod 
into on-screen 
buffer 107 is ali 
graphics data 
off-screen areas 
106 is a convention 
buffer 107 is cons 
memory devices (1 

FIG. 2 is a more 
VGA controller 105. The 
controller 105 include video 
memory (frame buffer) control 
control circuitry 202, video 
video [back-endl backend 
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memory in which video or 
either the on-screen or 
embodiment, display unit 
display device and frame 
dynamic random access 

functional block diagram of 
i circuitry blocks of VGA 
It-end video pipeline 200, 
ircuitry 201, CRT/window 
»w control registers 203. 
204 and graphics 



33 



40 



fback-endl backend pipeline 204 and 
205. VGA controller 105 further 1 
206 for exchanging instructions 
bus, such as local bus 103 in sya 
write buffer 207 and conventional 
allow CPU 101 to direcUy control 



s back-end pipeline 

[ludes a CPU interface 
I via a pa or VL 
\00, with CPU 101. A 
I controller 208 
[within £ranie buffer 
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so 



107 via memory control circuitry 201,' 

In the preferred embodiment of systciAlOO, CPU 101 can 
write video data and/or read and;wite 
buffer 107 via CPU interface 206. In pam Alar, CPU 101 can 
direct each pixel to the frame buffer usingtonc of two maps « 

depending on whether that pixel is a ^^deo PJ^el or a 
graphics puel. In the preferred cmbodimck each word of 
nixcl data ("pixel") is associated with one oitwo addresses, 
one which directs interpolation of the pixel A a video pixel 
through video front-end pipeline 200 and thVother which «, 
directs interpolation of the pUel as a flT ^^'^^ 
write buffer 207 and graphics coniroUcr 208. \s a conse- 
quence, cither video or graphics pixel data can tlten be input 
to CPU interface 206 from the PCI/VI bus 
"dual aperture" port as a function of the selected^address. 65 
Data which is input through the video port V^j_ 

free. In this case, video window controls 213 gencraes the 



required addresses to either the on-screen memory area or 
the dff- screen memory as a function of display location for 
the video window. In the preferred embodiment, window 
contr^s 213 generate addresses using the same video con- 
5 irol registers 203 used to control reuieval of the video in the 
[back-end] backeiW pipeline (i.e., the screen x and y position registers 
500 andSOl discussed below in conjunction with FIG. 5). 
When dab is being received through both the CPU interface 
206 and me VPORT 211 simultaneously, the data is inter- 
leaved intoXmemory with the two write bufFen 207 and 217 
buffering thkdata such thai neither stream is interrupted or 
forced into a vB^t state at the source component (i.e., bus 103 
or video sourc\ 104). 

It should be Vioted at this point that frame buffer 107 
includes at leastVwo different data areas or spaces to which 
data can be dircAed bv the given address (either CPU 103 
or controls 213 generated.) 

Each space can simultaneously 
store graphics or Mdeo data depending on the selected 
display configuratiox^ The on-screen area corresponds to the 
display screen; eacbXpixel rastered out of a given pixel 
location in the on-scresQ area defines a corresponding screen 
pixel. The off-screen kea is used to store data defining a 
window for selectivelA overiaying the data from the on- 
screen memory, fonts am other data necessary bv controller 
105. Fuither, as will be\ discussed [further] below, both graphics 
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and video data may be^ 
passed through vii 
graphics data is evi 
pipeline 205. 

According to the pr^i 
are alternate ways of 
video data from unifii 



I from frame buffer 107 and 
pipeline 204 while only 
through graphics backend 

sent invention, there 
and re^eving graphics and 
I buffer : 
[ write i 



i may be written into 
, the is rastered 
i video oiayback data 



For example, CPU luSmaj^ write a^static graphics back- 
grotmd into pan of the oti-sawn m^ory with the remain- 
ing "window** in the ODlscTBdn^^moiy area filled with 
playback video data. Tljpacln^video data can be either (1 ) 
live video data input fromihe VrOKT, (2) YUV (video) data 
written through interface 206 byVlPU 103; or (3) true color 
(5:5:5, 5:6:5. or 8:8:8) RGB g^phics data (for example 
animation graphics data) written in through either the 
VPORT or interface 206. Similari jL a playback video back- 
ground and a window of graphics i 
the on-screen area. In each of these c 
out as the display is without overlay: 
is passed through [the video playbaVk data is passed through] 

the video backend pipeliise 2IM as a fimcdon of display 
position by controls 202 and the g|[aphics data passed 
through the graphics backend pipeline 

Windows of data retrieved firom theW-screen memory 
can be retrieved and used to occlude aViortion of the data 
being rastered out of the on-screen memory. For example, a 
window of playback data can be storecain the off-screen 
memory and a &ame of static graphics data (either true color 
data or indices to CLOT 234) stored m the on-screen 
memory. In this case, the static graphics afle rastered out of 
the on-screen memory without interruptipn and passed 
through the graphics backend pipeline 2C6. vThe window of 
data in the off-screen memory is rastered ou&only when the 
display position for the window has been r^ched .by the 
display raster and is passed through video bamceod pipeline 
204. As discussed below, data &om the vmeo backend 
pipeline 204 can then be used to selectively omdude (over- 
lay) the dau being output firom the graphics backend pipe- 
line 205. A window of static graphics data (tri^ color or 
indices to the CLOT 234) can be stored inpff-screen 
memory and used [of] to 

oveday playback video o^m the 

on-screen memory[, the]. The 

playback video data is |assed 



[hrough ihe video backcnd pipeline 204 and ihe window of 
siaticteraphics daia is passed through the graphics backend 
pipeline 205. 

Bit block transfer (BitBLT) circuitry 209 is provided to 
allow Blocks of graphics daia within frame bulTcr 107 to be 5 
'j-ansfeAcd, such as when a window of graphics data is 
moved on the display screen by a mouse. Digital-to-analog 
corverteJ^DAC) circuitry 210 provides the requisite analog 
signals foiviriving display 106 in response to the receipt of 
cither vide\ data from video [back-end] backend 

A pipeline 204 or 

graphics dataVom [back-end pipe Une] backend P*P«^^« 

In implemcnimg the operations discussed above, video 
front-end pipeline 200 can receive data from two mutually 
exclusive input pAhs. First, in the "playback mode," play- 
back (non-real timi) data may be received via the PCI bus u 
thj-ough CPU intermcc 206. Second, in the "overlay emu- 
lation mode" cithcr\real-dmc or playback video may be 
received through the Video pen interface 211 (in system 100 
video port interface ill is coupled to bus 109 when real- 
time data is being received). The selection of video from the 20 
PCI bus or video from wdeo port interface 211 is controlled 
by a multiplexer 212 umicr the control of bits stored in a 
video front-end pipeline cVntroi register within video control 
registers 203. In the playback mode, either CPU 101 or a 
PCI bus master control linAthe PQ bus provides the frame 25 
buffer addresses allowing Aidco front-end pipeline 200 to 
map data into the frame buflfer separate and apart from the 
graphics data. In the overiay dpuladon mode, overlay input 
window controls 213 rcceivJp framing signals such as 
VSYNC and HSYNC, tracld these sync signals with 
counters to determine the start if each new frame and each 
new line, generates die requiredladdresses for the real-time 
video to the frame bufifer space u&ng video window position 
data received from window controls 222 (as discussed 
above, in the preferred embodiment, video data is always 33 
retrieved from either the on-scre«L 
and passed through video back-exyaTi 
tion of display posidon) and thiul 
controls 222 is used to boihyhiti 
retrieve data UierefromDJ.In siperi 
control windows are controlled pv tl 
control the [back-end] backenU 
video pipdini 

uisite counters and compaiatom 

overlay input video control drcmti 
Video front-end pipeline 2iA- 

circuitry 214 that is operable to tnind|ate 

data into an 8-bit formal and then rack 

encoded words into a single 32-bit wor 

written into the video frame buffer space 

105. Conversion circuitry 215 is operame 

555 data received from either the CPU li»e 

PCI bus or VPORTl/F 211 into YCtCb (Y 

encoding by encoding circuitry 214. Coc 

215 allows graphics data (for example in 

formal) to be intnxiuced through the VP 

data to be converted, packed and stored in 
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•screen memory 
le 204 as a func- 
ition data from 
memory and 
input video 40 
sters which 

^gh the req- 
intemal to 

includffs encoding 45 
16-bit YUV 422 
four such 8-bit 
d which is then 
of frame buffer 
convert RGB 50 

206 and the 
) data prior to 
iversion circuitry 
5:5:5 or 5:6:5 
>RT or graphics 55 
JV format in 

the off-screen memory space by CPU 10k For a more 
complete description of encoder 214 and the associated 
decoder 225 of video pipeline 204. reference ismow made to 
incorporated copending coassigned applicaricm Ser. No. 60 
08/223,845. The selection and control of iro encoding 
circuitry 214 and conversion circuitry 215 is iiAplcmented 
through m^lltiplexing cirouitries 212 and 216, eacn of which 
are controlled by bits in the video control ^egisteli^Fmall)^ 
video front-end pipeline 200 includes a write buffer/FIFO 65 
217 which in one embodiment acU as a write buffer and in 
an alternate embodiment acts as a FIFO for dxA video 
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backcnd pipeline 204. In embodiments where buffer 217 acts 

as a wile buffer for then Y, zooming on the [back-end,] backend, as 

discusscij below is by replication. In embodiments where 

buffer 2lV operates as a FIFO, then the VPORT and from 

and end colpr conversion by converter circuitry 215 are not 

used for wrtiing data to frame buffer 107. 

Memory control circuitry 201 includes an arbiter 218 and 
a memory interface 219. Arbiter 218 prioritizes and 
sequences requdsts for access to frame buffer 107 received 
from video front^end pipeline 200, graphics controller 208 
and bit block tra^fer circuitry 209. Arbiter 218 further 
sequences each ofVhese requests with the refresh of the 
display screen of diteplay 106 under the control of CRT 
controller 202. Memow interface 219 controls the exchange 
of addresses, data, ana control signals (such as RAS, CAS 
and read/write enable) \o and from frame buffer 107. 

CRT control/video wmdow control circuitry 202 includes 
the CRT controller 220\ window arbiter 221, and video 
display window controls u22. CRT controller 202 controls 
the refresh of the screen ox display 106 and in particular the 
rastering of data from frame buffer 107 to display unit 107 
through DAC 210. In the *eferred embodiment, CRT con- 
troller 220, through arbiter f 18 and memory interface 219, 
maintains a constant streamW graphics data into graphics 
backeod pipeline 205 from\ memory; video or playback 
graphics data is rastered out ^y when a window has been 

as determined by display 
itrols 222 (see FIGS. 3 and 
' controller 220. As will 
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reached by the display rasi 
position controls of window 
5 and accompanying text) ax 
be discussed in further detail 
within the display according 
invention is controlled in p; 
Video [back-end] backeni 
. pipeline 

ics video data defining a di! 
or off-screen spaces in frame 
fint-in/first-out memories 
where buffer 217 is acting as 
embodiment, each FIFO receive 
display line of data being gen 
display screen. For example, for a 
and n+1 in memoiy (although not 
the display) for the display window, 



display of windows 
frinciples of the present 
iiry 202. 

ves a v^jindow of graph- 
the on-screen 
ugh a pair of 
(in embodliments 
B). In the preferred 
data for every other 
for display on the 
f adjacent lines n-1 
[y adjacent on 
223 receives the 



pair 



data defining window display line n-l while FIFO 224 
receives the data defining window displiy line n+l. When 
buffer 217 is used as FIFO B, writes throijsb video front end 
pipeline 200 are made through write mjffer I 207 and 
multiplexer 235. Alternatively, if buffer 2317 is used as write 
buffer n, then FIFO B is not imolementedond only a single 
stream is processed by video [back-endl backend 

^ piiline 204 (no Y 

interpolation is performed and Y expansion is by replica- 
tion). As will be discussed further below ^assuming both 
FIFO A and FIFO B arc being used), one ol more display 
lines, which falls between line i>-l and lineWl, may be 
selectively generated by interpolation. Dccodcrtcircuitry 225 
receives two 32-bit packed words (as encodea by encoder 
214), one from each adjacent scan line in memory, from 
FIFOs 223 and 217. Each 32-bit word, which rcdrescnts four 
YCiCb pixels, is expanded and error diffused py decoder 
225 into four 16-bit YCrCb pixels. In modes wbsn video 
Hatfl is Stored in the frame buffer in standard 555 HCB or 16 
YCrCb data formats, decoder block 225 is bypi 

(Back-endl Backend 

video pipeline 204 further includes a V inter- 
polator 226 and X interpolator 227. In the preferred embodi- 
ment, during Y zooming (expansion) Y interpolate 226 
accepts two vertically adjacent 16-bit RGB or YCrCb toels 
from the decoder 225 and calculates one or more resampled 
output pixels using a four subpixel granularity. X ii 



^11^27 during X zooming (expansion) accepts horizon 
tally adjacent pixels from the Y interpolator 226 and calcu 
latcs onaor more resamplcd output pixels using a four 
subpixcl jranulariiy. For data expansion using line replica- 
tion, Y inkrpolator 226 is bypassed. Y interpolator 226 and 
X inicrpoLor 227 allow for the resizing of a video display 
window bling generated from one to four times. 

The out^t of X interpolator 227 is passed to a color 
convener 2& which converts the YCrCb data into RGB data 
for deliver7\to output multiplexer 304. To reiterate, if 
graphics data V passed through.video pipeline .converter 228 
is not used. 

[Back-endt Backend 

vidao circuitry 204 further includes pipeline 
control circuitry\229, overlay control circuitry 230 and 
output multiplexer 231. Pipeline control circuitry 239 con- 
uols the reading 4f data from video FIFOs 223 and 217. 
controls the generation of intcrpolauon coefficients for use 
by X and Y intcrpSlators 226 and 227 to resize the video 
window being pipelined, and times the transfer of data 
through the pipelinl. Overlay control circuitry 230 along 
with control circuity 202, controls the output of data 
through output multiplexer 231. including the overlay of the 
video window overdo eraohics data output through the 

graphics [back-endl\ackend . . -.a 

pip^ine 205. A pUel doubler is provided 
to double the number 4f pixels being generated such that a 
1280x1024 display cadbc driven. 
Graphics [back-end J backend 

pipeline 203 includes a firsi-in/first- 
" iniroUer 233. and color look-up 
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out memory 232» aitril 
table 234. Each 32-bit wj 
is serialized into either] 
8-bil words, typically 
attribute code, are seti: 
16-bit and 24-bit word 
serialized, those words. 
230. Attribute coniroll* r 2 J3 
and underlining open lio is 
output from attribute c on roll 
used to index GLUT 23 1 
24-bil words of pi^d di 
each index. When vi?-* 
graphics backend pipe! 
GLUT 234 is bypassed 
The eight bit pseudocolor p 
controller 233 are also sent l 
preferred embodinwnt, data is 
on-screen memory through ~ 



itput from graphics FIFO 232 
16-bit or 24-bit words. The 
_ of an ASGn code and an M 
ibuic controller 233. When 
typically color data, are 
sent d^jfcctly - overlay controls 
fornjs such tasks as blinking 
in texymodes. The eight bits 35 
are pseudo-color pixels 
234 preferably outputs 
^output multiplexer 231 with 
is being pipelined through 
[from the on-screen memoiy, 40 



Js [arel output from attribute 
^overlay controls 230. In the 
atintxously pipelined from 
lies [back-end] backend 

pipeline 205 45 
231. Window data from 
retrieved from memory 
_ pipeline 204 when a 
words, when a window 
.control bits set by CPU 



so 



to the inputs of output mullipli 
off-screen memory however is < 
and pipelined through video back 
window is being displayed. In ott 

has been reached, as deiennined L^. . 

101 in VW comiol rcgisien 222,\video wmdow display 
controls 222 generate addresses to dpirieve the correspond- 
ing data from the off-screen memori space of frame bi^cr 
l(n. Preferably, video FIFOs 223 anb 224 are filled before 
the raster scan aciuaUy reaches die disblay window such ttat 
the initial' pixel data is available iimcdiaiely once the 
window has been reached. In order toVinsure that graphics 
memory data continues to be provided *>-^-'^ 
pipcUne 205. video window display dpntrols 222 steal 
page cycles between page accesses to thfc grajAics niemOTy. 
It should be noted that once the windo^A has been reached 
the frequency of cycles used to rciriiye window data 
increases over the number used to fill the \ 
outside a window. When the frequency 
accesses increases, video window display 
arbiter 221 preferably "steal" cycles from pag 
used to write data into the frame buffer. 



55 



60 



> FIFOs when 
' window page 
L controls tU/ 

t cycles being 
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. 3 is a more detailed functional block diagram 
sizing the circuitry controlling the overlay of data 
kraphics pipeline 205 with window data from video 
pipcliAe 204. As discussed briefly above, the inouts to output 
multipfl^xer 231 are data from video [back-end] backend 

, pipeline 204 

(pixel ddublcr 237), 16 or 24-bit color data directly from 
graphics Wk-end pipeline 205 serializer 236 and 24.bit 
color data)from the color look-up table 234. The output of 
data to DAU 210 through output multiplexer 231 is con- 
trolled by a\atch 301 clocked by the video clock (VCLK). 
The remaini^ circuitry shown in RG. 3. which will be 
discussed in nmher detaU below, provide the necessary 
control signals\to the control inputs of output multiplexer 
231 to select bdtween the video and graphics pipelines. 

The graphics reeudo-pixels output from attribute control- 
ler 233 and the loVbii or 24-bit graphics or video data output 
directly from scriadizer 236 are provided to the inputs of 
color comparison Jircuitry 302. Also input to color com- 
parison [circuit] ciAcuitry 

302 We 16 or 24-bil overlay color key bits 
stored in overlay cotor key register 303. Overlay color key 
register 303 resides Wthin the address space of. and is 
loaded by, CPU 101. ^Depending on the mode, color com- 
parison circuitry 302 cWpares selected bits from the over- 
lay color key register 303 with either the 8 bits indexing 
look-up cable 234 in the itolor look-up table mode (pseudo- 
color mode) or the 16-bitft(24-bits in the alternate embodi- 
ment) passed directly fr^mverializer 236. It should be noted 
that in the illustrated embodlmeQt, oveday color key register 
303 holds (24-bits of] 24 /\\ 

of ov^aAcol^r key bits, eight each for 



30 red, green, and [blue/inxl 

color key biu compared\ 
provided in Table I: 



/index 

i specific overlay 
input g^a})tiics data are 
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MODE 


overiaJ 




)Ijoi^key bits compared 


CLin' 








Index 








5:5:5 


Re(k4:0> 




mcAm y Bhie<4:a> 


5:6-J 


Red<4:Q> 


g) 


|eiK5KbL/^ Bhie<4:a> 


8:8:8 




Gr 


9pa<;:fiA Bliie<7:Q> 



As shown in FIG. 4A, a first emlMdimem of color 
comparison dicuitry 303 perfonns the comparisons set forth 
in Table I as a set of XNOR operaiioDS in sories with an AND 
operation. RG. 4A depicts first comparison circuitry 400 for 
comparing the 8-bits of graphics pixels \received in the 
look-up table mode from attribute comroller 233 with the 
8-bit blue/index overiay key bits being held in ovetiay key 
register 303. Second comparison drcuitxy 4(11, poforms the 
required comparisons of Table I for the 16-b« data or 24-bit 
received from serializer 236, in, either a 5:5:515:6:5, or 8:8:8 
format An overiay register 402 includes a urit loaded by 
CPU 101 which is used by a selector 403, deftnding on the 
mode, to select for output, either the result onthe compari- 
sons being made by comparison circuitry 40diin the color 
look-up table mode or the results of the compfiunsons being 
made by comparison dicuitry 401. In the iUustrasd embodi- 
ment, color comparison drcoitry 303 processeAd^ta a 
pixel-by-pixel basis and is resynchionized wii 
graphics [back-end] backend 

pipeline 205 and the video 
pipeline 204 by having its ou^nxu latched to the vi( 
(VCLK) by latches 404. 

The output of color comparison circuitry 303 is 
the "K** control input of overlay control raultipl 
65 The "F' control input to multiplexer 304 is pro' ' 
pixel position comparison circuitry 305. The data inpi 



iboth the 

;-end 
clock 



to 
304. 
from 
to 



multiplexer 304 are coupled to an 8-bit overiay OP < 
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(OOC) register 306. The ouipul of multiplexer 304 is used 
Vnc control input to output multiplexer 304, which along 
1 a single bit set by CPU 101 into output conuol register 
V selects which of the data received at the data inputs of 
._..jlexer 231 will be ouipul to DAC 210. 5 
PiAl position comparison circuitry 305 includes three 
inputs\oupled respectively to video window 1 po^^^on 
control dVcuitry 308. CRT position control circuitry 309 and 
video wi^ow 2 position control circuitry 310. In the illus- 
trated emWidimcnt, CRT position controller 309 is located 
within CRT\ontrollcr 220 while video window 1 position 
control circiAry and video window 2 position control cir- 
cuitry 310 arelocated within video display window controls 
222 (FIG. 2V CRT oosition control circuitry 309 includes 
counters ,^^;J^ , ^^^^ 

c the position of the current pixel being 

generated for dkplay. In the preferred embodiment. CRT 
position control Ircuitry 309 includes at least an x-pqsition 
counter which trkks the generation of each pixel along a 
given display linctnd a y-position counter which tracks the 
generation of eaclTdisplay line in a screen. The x-posilion 
counter may for Aample count pixels by counting each 
VCLK period bctvteen horiiontal synchronization signal 
(HSYNC) coniroluXg display unit 106. The y-position 
counter may for exaAple count each HSYNC signal occur- 
ring between each vertical synchronization signal (VSYNQ 
controlling the screcn\encration on display unit 106. FIG. 
4B is an alternate cmfeodimcni of the color comparison 
circuitry of FIG. 3. In attrst mode, 8 bits [are] 

J^..*^ from attribute 

controller 233 are p 

406. Comparator 
an 8-bit color key 
received 8-bits equa 
tor 406 go es active , 
T^SITGR is high (i 
consequently high) 
406 is gated through 

410 is passed to A> 
from the pixel comparis 
gate 411 goes directly 
231 (in this embodiment 

are eliminated). Thus, 

from attribute controller 233 
matches the 8-bit color key 1 
reached as determined by pixel 
the pixel data output from video 
through selector 231. 

In a second mode. 16 bits are 
The eight LSBs are passed throul 
comparator 406 and tto eight MSBi 

407. Control signal 16B1TNG is sei 
equal key 1 in color register key 408 
key 2 in color key register 408, the 
tors 406 and 407 are active (high). 1 

411 then goes high when the output 
circuitry 305, which is coupled to the 
selector 231, goes high. Thus, when 
output from serializer 236 of graphics 
the 16-bii color key (keys 1 and 2) and 
reached, the output pixel data from video 
passed through selector 231. 

RG. 5 is an expanded functional block cnagram of the 
video window position control circuits 308 and a corre- 
sponding portion of the gating of pixel positibn compare 
circuitry 305. Each position control circuit Blfl/312 is 
coupled to a screen position x-registcr 500 anS a soecn 65 
position y-registcr 501, and includes a screen V-position 
counter 502, and a screen y-posiiion counter 50i In the 



( multiplexer to comparator 
i received eight bits with 
register 408; when the 
^e output of compara- 
. control signal 



\ output 

! output 



gate 409 is 
tm comparator 
It of AND gate 
rith the output 
output of AND 
)\ input of selector 
and register 306 
graphics pixels output 
if graphics backend 205 
the window has been 
»mpaiison circuitry 305, 
lackend 204 are passed 

dfrom serializer 236. 
multiplexer 406 to 
I to comparator 
iigh. When the LSBs 
i the 8 MSBs equal 
uu from compara- 
titputof ANDgate 
i pixel comparison 
p** control input of 
^ 16-bit pixel data 
1205 matches 
dow has been 
nd 204 are 
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^'diment, registers 500 and 501 arc located 

cnrrPcV. A' V '^""""^ ^03. For the window 

rote"' "^'^^^^ --itry 

308 o J310, registers 500 and 501 are loaded with a valu^ 

ZTJT^ " f"^ ' '"^^^ °^ pi-el in the 

rS S sooir^^ ^^'^ ^^^"^ Screen 
embodimW are loaded by CPU 101. screen x posi™ 
^ counter 5S counts down from the value held inTeen 
x-reg,ster 5^0 with each video clock when P is high for each 
display hne aL resets with each display horizont^ syncS 
nuation s^gnaiW YNC) (Note that when P is high Tc^ 
count notches tXe position coumXSo^^ 
503 counts dowX from the value set into screen y-r^gis^ 

Inh T M^""^ ^"SYNC) at the stan of 

each display hne Ad resets with each VS YNC at the stm of 
each new screen (^e position counters a« allowed to ^ 
only when they mich their penpective COT). Ttc coZ 
v^ues :n the counteL of COTposition control circuitJ^S 
are compan«l puel\by pixel with the counts in Tcrcea 
x-posmon counter 5(b and screen y-position 503 of each 
video wmdow posiuoA control ciicuitiy 308 and 310. When 
bo^ the X and y cou^ m the counten of COT position 
conrn,! circuitry 309 math the conesponding x and y counts 
m respective counters sfc and 503 of either video window 
1^0, the control signal P to multi- 
acUj^on of control signal P 
Ofl^spky 106 has reached the 
[winJowJ window, 



control circuitry 308 or 
plexer 304 is activated, 
indicates that the raster sc 
position of a pixel within 



and data from video 
Wjngon the value being 
regSt&>^06 and the K 
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pipeline 205 may be pi 

held in overlay OP Code 
33 control inputs to multiplex! 
A 4-bit OP Code loaded 

Code register 306 in conjuni 

applied to the "P" and "IC* coi 
40 control the presentation of - 

illustrated embodiment) con 

input to output muliiplcxCT^^ , „ ^ _ 

output mtiltiplexer 231 re^JV^Ta bA&om overlay mode 
register 402 (HG. 4), as loaded AcPU 101. In the 
iHustnued cmbodimcnu the selection bUeen the streams 
from the graphics and video backends atW 0.1,2 inputs to 
output multiplexer 304 in response to the Wnals presented 
at the corresponding canirol inputs "A A and "B" is in 
accordance with Table 0 



'U 101 intojoveriay OP 
with the CQfttroi signals 
tom(iltiplexer304 
(as^iaed high in the 
the "B" control 
other ("A") input to 



45 



Control Input A 


Control Input B 


Selected Strkm 


0 


0 


Gmphics or \ 
video ptxeU \ 
from gfnyhig \ 


I 




pipeline 205 \ 


0 


Gr^idcs pixels \ 
flora CUJT 234ft 




1 


input I y 


0 
1 


Video or \ 


1 


gnpUct from \ 
video hackead \ 
204 \ 
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The OP Codes used in the iUustrated embodiment, , 
effective overlay and the corresponding inputs to the cont 
mputs of multiplexer 304 are listed in Table m (active sti 
is assumed{:]): 
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Civcrlsv Oo Code 


NtuliiplcxcT 307 






Ra»isicr 309 Value 


Control Inputs 


EiTca 




\ ° 


N/A 


Pixels passed 
only from 
grsphics pipeline 
205 


5 




Irpul P active 


Point pixels from 
video bockcnd 








pipeline 204 only 


10 






inside video 






window VDW. 




8 \ 


Inputs K and P 


Paint from video 






active 


backend pipeline 
204 window VOW, 








when color key 


IS 






matches 


C \ 


laputs K active 


Paint from video 
backend pipeline 
204 if color key 
tnatches 





In the illusiratedBmbodiment, if a Oh is written into OOC 
register 306 by CPU 101, only pixels from graphics pipeline 
205 arc pipelined thk}ugh multiplexer 304. In this case any 
. signals applied to thdP and K control inputs to multiplexer 
304 have no effect (i.A, will not result in a high output &om 
multiplexer 304). In wSt illustrated embodiment, if an Ah is 
written into OOC regiger 306, pixels from video pipeline 
204 will be passed to D^C-210 only when pixel position 
comparison circuitry 30xaeterntines that the raster scan has 
reached a pixel in the winidow and hence the control signal 
going to the P input of mimiplaScer 304 has been activated 
If on the other hand,, an 8h i\ wHtten into OOC register 306, 
data is passed throu^ outoAri mltiDlexCTg l to DAC 210 
Ipommukcm circuitry^3Q5 determines 
chea a pixel on the di^ay screen 
id colon compare circuitrA302 has 
coming nata from graphics pipeline 
f color ny held in overiay oblor key 
, the dsa from video pipieline 204 
^ben botivthe P and thj>4c inputs to 
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when pixel position^ 
that the raster scan 1 
within the window 
determined that the il 
205 matches the overl 
register 303; In this ( 
is passed to DAC 210 ^ 

multiplexer 304 are acti^. Finall when ap^dpCode of C is 
programmed into OOC register 3(l6^d^afrom video pipe- 
line 204 is passed if the J upogggg data &om graphics 
pipeline 205 matches the ovoiay (T)lQr key held in overiay 
color key register 303. In this easel the activation of the K 
control input activate the output of iimltiplexer 304 to switch 45 
the input of multiplexer 231 to pass tpe corresponding video 
pixels. 

Display control circuits embodyiib the principles of the 
present invention have substantial advantages over the prior 
ait In particular, output control circuits built in accordance 
with the principles of the present inwidon allow for the 
flexible display of both graphics anm video on the same 
screen. In particular, pixel position comparison circuitry 305 
along with video window position control circuits 308 and 
310 and CRT position control circuitry allow for [a ] 

one 

or more windows from off-screen «««>n"',^ 
in specific areas of a display screen to thc\xclusi^nof ^any 
simultaneously generated data from on-sc 
ihcr, color comparison circuitry 302 ope 
uon with an overiay color key written into oV 
register 303 aUows window data to be 
display screen, to the exclusion of any co 
ated aranhics data, without the need f^ 
^Ssfuon the window. FmaUy. theW of the 

'gSS daufrom the graphics pipeline 205 toVont^l Je « 
oSJut overlay provides additional advamagesY°« the 
video data can be subject to graininess and noise.) 
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I memory. Fur- 
pg in conjunc- 
iay color key €0 
nted on the 
tly gencr- 
ase X- and 



AUhou; 
been descri' 
changes, 



wichout depanii 
as defined by 
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cm invention and its advantages have 
it should be understood that various 
[tions\and alterations [^3^] can 

be made herein 
the spirit and scope of the invention 
oded claims. 



